LANGUAGE UNDERSTANDING

What do Large Language Models Learn about Scripts? (2022.starsem-1)

Challenge:	Existing pre-trained language models are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios.
Approach:	They propose a knowledge-injected pre-trained language model that can be transferred to both natural language understanding and generation tasks.
Outcome:	The proposed model significantly outperforms baselines across the board in e-commerce scenarios.

Copied to clipboard

Abhilasha Sancheti, Rachel Rudinger

Deep Learning for Natural Language Inference (N19-5)

Challenge:	Script Knowledge is important for language understanding but expensive to produce manually and difficult to induce from text due to reporting bias.
Approach:	They propose a pipeline-based script induction framework which can generate good quality ESDs for unseen scenarios.
Outcome:	The proposed framework produces good quality ESDs for unseen scenarios, but manual evaluation shows there is room for improvement.

Copied to clipboard

Samuel Bowman, Xiaodan Zhu

Knowledge-Augmented Methods for Natural Language Processing (2022.acl-tutorials)

Challenge:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning.
Approach:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models.
Outcome:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning.

Copied to clipboard

Chenguang Zhu, Yichong Xu, Xiang Ren, Bill Yuchen Lin, Meng Jiang, Wenhao Yu

Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study (2024.naacl-long)

Challenge:	Knowledge in natural language processing (NLP) is a rising trend especially after the advent of large scale pre-trained models.
Approach:	This tutorial introduces the key steps in integrating knowledge into natural language processing (NLP) it introduces knowledge grounding from text, knowledge representation and fusing.
Outcome:	This tutorial introduces the key steps in integrating knowledge into natural language processing including knowledge grounding from text, knowledge representation and fusing.

Copied to clipboard

Yinghao Li, Haorui Wang, Chao Zhang

Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval (2026.acl-srw)

Challenge:	Large Language Models (LLMs) have shown remarkable proficiency in language understanding and have been successfully applied to a variety of real-world tasks through task-specific fine-tuning or prompt engineering.
Approach:	They propose a task that challenges LLMs to identify the locations of mines based on numerical clues provided by adjacent cells.
Outcome:	The proposed task requires an understanding of each cell’s state, discerning spatial relationships between clues and mines, and strategizing actions based on logical deductions drawn from the arrangement of the cells.

Copied to clipboard

Matei Benescu, Ivo Pascal de Jong

Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding (2024.naacl-tutorials)

Challenge:	Recent studies show that similarity is a short-sighted interpretation of relevance . LLM-based Relevance Judgment Systems (LLM-RJS) can overcome this limitation .
Approach:	They propose that LLM-Based Relevance Judgment Systems can overcome short-sighted interpretation of relevance by embedding similarity instead of similarity.
Outcome:	The proposed methods outperform Neural Embedding Retrieval Systems by overcoming similarity limitation.

Copied to clipboard

Parisa Kordjamshidi, Qiang Ning, James Pustejovsky, Marie-Francine Moens

Representation Learning

Challenge:	This tutorial provides an overview of cutting edge research on spatial and temporal language understanding.
Approach:	This tutorial provides an overview of cutting edge research on spatial and temporal language understanding.
Outcome:	This tutorial provides an overview of cutting edge research on spatial and temporal language understanding.

A Hybrid Approach to Scalable and Robust Spoken Language Understanding in Enterprise Virtual Agents (2021.naacl-industry)

Copied to clipboard

Ryan Price, Mahnoosh Mehrabani, Narendra Gupta, Yeon-Jun Kim, Shahab Jalalvand, Minhua Chen, Yanjie Zhao, Srinivas Bangalore

Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)

Challenge:	Spoken language understanding (SLU) extracts the intended mean- ing from a user's utterance.
Approach:	They propose a framework for intent and entity extraction utilizing a hybrid of statistical and rule-based approaches.
Outcome:	The proposed framework can be deployed quickly for a large class of EVA applications with little need for human intervention.

Copied to clipboard

Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, Trevor Cohn

NumeroLogic: Number Encoding for Enhanced LLMs’ Numerical Reasoning (2024.emnlp-main)

Challenge:	Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively.
Approach:	They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models.
Outcome:	The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation.

Copied to clipboard

Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle

Numerical Reasoning

Challenge:	Language models struggle with numerical and arithmetical tasks, such as multiplying 3-digit numbers.
Approach:	They propose a method to include the count of digits before each number instead of “42”.
Outcome:	The proposed format improves the reasoning process before generating the actual number.

Toward Diverse Precondition Generation (2021.starsem-1)

Copied to clipboard

Heeyoung Kwon, Nathanael Chambers, Niranjan Balasubramanian

Challenge:	a typical goal for language understanding is to logically connect the events of a discourse, but connective events are not described due to their commonsense nature.
Approach:	They propose a system that generates unique and diverse preconditions by using an event sampler, candidate generator, and post-processor.
Outcome:	The proposed system can generate unique and diverse preconditions without training on diverse examples.

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP (2023.emnlp-main)

Copied to clipboard

Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed

Dialogue Systems & Conversational Agents

Challenge:	Our study examines ChatGPT’s performance on Arabic languages and dialectal varieties.
Approach:	They conduct a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets.
Outcome:	The proposed model outperforms smaller models on Arabic dialects compared to GPT-4's Modern Standard Arabic and Dialectal Arabic (DA)

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually) (2020.emnlp-main)

Copied to clipboard

Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, Samuel R. Bowman

FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)

Challenge:	Pretraining on self-supervised linguistic tasks is effective for learning features helpful for language understanding, but it requires more data to learn to prefer linguistic generalizations over surface ones.
Approach:	They propose a set of 20 ambiguous binary classification tasks to test whether a pretrained model prefers linguistic or surface generalizations.
Outcome:	The proposed model can learn to represent linguistic features with little pretraining data, but requires far more data to learn to prefer linguistic generalizations over surface ones.

Copied to clipboard

Rinat Khaziev, Usman Shahid, Tobias Röding, Rakesh Chada, Emir Kapanci, Pradeep Natarajan

Dialogue Systems & Conversational Agents

Challenge:	Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error .
Approach:	They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions.
Outcome:	The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant.

The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)

Copied to clipboard

Nathan Schucher, Siva Reddy, Harm de Vries

Data Contamination: From Memorization to Exploitation (2022.acl-short)

Challenge:	Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks.
Approach:	They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations.
Outcome:	The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations.

Copied to clipboard

Inbal Magar, Roy Schwartz

Challenge:	Pretrained language models are typically trained on web-based datasets that are often "contaminated" with downstream test sets.
Approach:	They propose a method to pretrain BERT models on Wikipedia and labeled downstream datasets and fine-tune them on the relevant task.
Outcome:	The proposed method compares models on Wikipedia and labeled downstream datasets on two models and three downstream tasks.

TextWorldExpress: Simulating Text Games at One Million Steps Per Second (2023.eacl-demo)

Copied to clipboard

Peter Jansen, Marc-alexandre Cote

Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)

Challenge:	Text-based games are limited in speed due to legacy tooling and limited implementations.
Approach:	They propose a text-game simulator that reimplements three common text game benchmarks and runs three orders of magnitude faster than their original implementations.
Outcome:	The proposed simulator increases throughput by three orders of magnitude, reaching over one million steps per second on common desktop hardware.

Copied to clipboard

Yen-Ling Kuo, Boris Katz, Andrei Barbu

Explainability & Interpretability in NLP

Challenge:	a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study.
Approach:	They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains.
Outcome:	a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not.

“We Demand Justice!”: Towards Social Context Grounding of Political Texts (2024.emnlp-main)

Copied to clipboard

Rajkumar Pujari, Chengfei Wu, Dan Goldwasser

Challenge:	Political discourse on social media often contains similar language with opposing intended meanings.
Approach:	They propose to characterize the social context required to fully understand political discourse . structured models outperform larger models on both tasks, but still lag behind human performance .
Outcome:	The proposed models outperform larger models on both tasks but lag behind human performance.

Limits for learning with language models (2023.starsem-1)

Copied to clipboard

Nicholas Asher, Swarnadeep Bhar, Akshay Chaturvedi, Julie Hunter, Soumya Paul

Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs (2026.eacl-long)

Challenge:	Recent studies show that large language models fail to capture important aspects of linguistic meaning . authors argue that LLMs cannot learn fundamental semantic properties defined in formal semantics .
Approach:	They propose a theoretical explanation for some of the observed failings of large language models . they show that LLMs cannot learn certain fundamental semantic properties .
Outcome:	The proposed model fails to learn semantic entailment and consistency as defined in formal semantics, the authors argue . their model fails on tasks that require engorgements and deep linguistic understanding, they argue - but not on universal quantification.

Copied to clipboard

Hussein Abdallah, Ibrahim Abdelaziz, Panos Kalnis, Essam Mansour

Structured Data in NLP

Challenge:	Traditional KGQA assumes a closed world where answers must exist in the KG, limiting real-world applicability.
Approach:	They propose a system that combines a pre-trained GNN and an LLM for open-world QA.
Outcome:	The proposed system outperforms existing LLM–GNN systems on standard benchmarks and GLOW-BENCH, achieving up to 53.3% and an average 38% improvement.

JI2S: Joint Influence‐Aware Instruction Data Selection for Efficient Fine‐Tuning (2025.emnlp-main)

Copied to clipboard

Jingyu Wei, Bo Liu, Tianjiao Wan, Baoyun Peng, Xingkong Ma, Mengmeng Guo

How effective is BERT without word ordering? Implications for language understanding and data privacy (2021.acl-short)

Challenge:	Prior selection strategies score samples using generalpurpose LLMs, leveraging their strong language understanding but introducing inherent biases that misalign with the target model’s behavior and yield unstable downstream performance.
Approach:	They propose a framework that jointly models marginal and combinatorial influences within sample groups and evaluate them on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons.
Outcome:	The proposed framework outperforms fulldataset training and strong baselines on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons.

Copied to clipboard

Jack Hessel, Alexandra Schofield

DuRE: Dual Contrastive Self Training for Semi-Supervised Relation Extraction (2024.naacl-long)

Challenge:	Ordered word sequences contain the rich structures that define language.
Approach:	They show that token representations and self-attention activations within BERT are resilient to shuffling the order of input tokens.
Outcome:	The proposed model is able to handle shuffled token representations and self-attention activations . the model can handle GLUE language understanding tasks with bag-of-words data .

Copied to clipboard

Yuxi Feng, Laks Lakshmanan

PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules (D19-3)

Challenge:	Existing document-level relation extraction methods require manual training and labeled data to obtain supervised learning.
Approach:	They propose a document-level relation extraction framework that integrates RE and text generation as a dual process.
Outcome:	The proposed framework significantly boosts recall and F1 score with comparable precision on two document-level RE tasks against several strong baselines.

Copied to clipboard

Youngsoo Jang, Jongmin Lee, Jaeyoung Park, Kyeng-Hun Lee, Pierre Lison, Kee-Eung Kim

Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)

Challenge:	a recent development of spoken dialogue systems has enabled deep learning to achieve state-of-the-art performance.
Approach:	They propose a Python-based domain-independent, open-source toolkit for spoken dialogue systems.
Outcome:	The proposed toolkit extends OpenDial's Java-based architecture and provides new functions for neural dialogue state tracking and action planning.

Copied to clipboard

Xudong Hong, Margarita Ryzhova, Daniel Biondi, Vera Demberg

Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023.acl-srw)

Challenge:	Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning.
Approach:	They used a script-based story to manipulate event B in a story which causally depends on a previous event A.
Outcome:	The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition.

Copied to clipboard

Julia Sato, Helena Caseli, Lucia Specia

Alignment Annotation for Clinic Visit Dialogue to Clinical Note Sentence Language Generation (2020.lrec-1)

Challenge:	Pre-trained language models have achieved remarkable results on several NLP tasks.
Approach:	They propose three new masking strategies for cross-lingual visual pre-training that focus on learning different linguistic patterns.
Outcome:	The proposed methods outperform the baseline model and achieve state-of-the-art accuracy on the Portuguese-English MMT task.

Copied to clipboard

Wen-wai Yim, Meliha Yetisgen, Jenny Huang, Micah Grossman

End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2 (2020.acl-main)

Challenge:	Despite advances in natural language processing, converting a clinic visit conversation into a clinical note is a largely unexplored area of research.
Approach:	They propose an annotation methodology that is content- and technique- agnostic while associating note sentences to sets of dialogue sentences.
Outcome:	The proposed method is content- and technique-agnostic while associating note sentences to sets of dialogue sentences.

Copied to clipboard

Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, Kee-Eung Kim

Dialogue Systems & Conversational Agents

Challenge:	End-to-end dialogue systems with monolithic neural architecture are often trained with input-output utterances without taking into account the entire annotations available in the corpus.
Approach:	They propose an end-to-end neural architecture for goal-oriented dialogue systems that addresses both challenges . they propose a modular architecture where modules are optimized individually .
Outcome:	The proposed system achieved the top position in the human evaluation task . it is based on a neural architecture that can be integrated with external systems .

Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data (2024.findings-acl)

Copied to clipboard

Haolong Li, Yu Ma, Yinqi Zhang, Chen Ye, Jie Chen

CBAL: Context-Based Agentic Learning for Speaker Diarization Segmentation Refinement (2026.acl-srw)

Challenge:	Large language models (LLMs) have shown excellent capabilities in language understanding, text generation and many other tasks, but struggle in complex multi-step reasoning problems such as mathematical reasoning.
Approach:	They propose to fine tune an open-llama-3B model to perform well on multi-step reasoning tasks via synthetic data.
Outcome:	The proposed model can reach a zero-shot pass@1 at 0.44 on the in-domain dataset and demonstrates certain generalization capabilities on the out-of-domain data.

Copied to clipboard

Odwitiyo Dutta, Dinesh K Vishwakarma

PromptPrism: A Linguistically-Inspired Taxonomy for Prompts (2026.findings-eacl)

Challenge:	Speaker diarization systems produce segmentation errors that degrade transcript readability and downstream applications.
Approach:	They propose a framework that refines segmentation boundaries in diarized scripts . they use a lightweight LLM agent to reason about merge decisions .
Outcome:	The proposed framework achieves 93.4% accuracy across 359 applied merges and reduces segment count by 6.1%.

Copied to clipboard

Sullam Jeoung, Yueyan Chen, Yi Zhang, Shuai Wang, Haibo Ding, Lin Lee Cheong

Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions (2023.findings-eacl)

Challenge:	PromptPrism is a linguistically-inspired taxonomy that enables prompt analysis across three hierarchical levels.
Approach:	They propose a linguistically-inspired taxonomy that enables prompt analysis across three hierarchical levels: functional structure, semantic component, and syntactic pattern.
Outcome:	The proposed taxonomy bridges traditional language understanding with modern LLM research . it improves prompt quality and improves model performance across tasks .

Copied to clipboard

Henrik Voigt, Jan Hombeck, Monique Meuschke, Kai Lawonn, Sina Zarrieß

Towards Unsupervised Language Understanding and Generation by Joint Dual Learning (2020.acl-main)

Challenge:	Existing language and vision models can be used for language understanding in 3D environments . however, existing models lack specific properties and biases that limit their performance .
Approach:	They propose a framework that uses a camera to generate images from different viewpoints and evaluate them in terms of their similarity to natural language descriptions.
Outcome:	The proposed model performs poorly on most canonical views and fine-tunes using hard negative sampling and random contrasting yields good results even under conditions with little available training data.

Copied to clipboard

Shang-Yu Su, Chao-Wei Huang, Yun-Nung Chen

Low-Resource NLP

Challenge:	Existing work exploits dual property between understanding and generation to improve performance of modular dialogue systems.
Approach:	They propose a dual supervised learning framework that exploits the dual property between understanding and generation.
Outcome:	The proposed framework improves both NLU and NLG performance by incorporating supervised and unsupervised learning algorithms.

Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021.naacl-main)

Copied to clipboard

Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu

Explainability & Interpretability in NLP

Challenge:	Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding.
Approach:	They propose a shortcut mitigation framework to suppress NLU models from making overconfident predictions for samples with large shortcut degree.
Outcome:	The proposed framework suppresses the model from making overconfident predictions for samples with large shortcut degree.

Text Segmentation as a Supervised Learning Task (N18-2)

Copied to clipboard

Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, Jonathan Berant

An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)

Challenge:	Existing datasets for text segmentation are small in size and do not represent the natural distribution of text in documents.
Approach:	They propose a large dataset for text segmentation that is automatically extracted and labeled from Wikipedia and develop a model based on this dataset.
Outcome:	The proposed model generalizes well to unseen natural text.

Copied to clipboard

Hongchao Fang, Pengtao Xie

Generalization in Instruction Following Systems (2021.naacl-main)

Challenge:	Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately.
Approach:	They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training .
Outcome:	Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP.

Copied to clipboard

Soham Dan, Michael Zhou, Dan Roth

Challenge:	Understanding and executing natural language instructions in a grounded domain is one of the hallmarks of artificial intelligence.
Approach:	They propose a learning strategy that involves data augmentation to improve the model's performance.
Outcome:	The proposed learning strategy outperforms state-of-the-art models in the blocks world domain while satisfying our expectations much better.

Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow (2024.emnlp-industry)

Copied to clipboard

Tian Guo, Emmanuel Hauptmann

Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations (2026.findings-eacl)

Challenge:	Quantitative investing relies on extracting quantitative features or signals from various data sources including market prices, economic indicators, financial text, etc.
Approach:	They propose to integrate LLMs’ token-level embeddings into a forecasting module and compare their results to those of encoder-only and decoder-based models.
Outcome:	The proposed model outperforms conventional sentiment scores on multiple investment universes and is based on encoder-only and decoder-based models.

Copied to clipboard

Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang, Hen-Hsen Huang, Hsin-Hsi Chen

When Do You Need Billions of Words of Pretraining Data? (2021.acl-long)

Challenge:	Recent advances extend language understanding beyond text to speech, enabling unified reasoning across modalities.
Approach:	They construct and release a speech-augmented benchmark based on Global MMLU Lite and a data set spanning English, Chinese, and Korean.
Outcome:	The proposed model is robust to demographic factors but sensitive to language and option order, suggesting that speech can amplify structural biases.

Copied to clipboard

Yian Zhang, Alex Warstadt, Xiaocheng Li, Samuel R. Bowman

Selecting Key Views for Zero-Shot Entity Linking (2023.findings-emnlp)

Challenge:	Pretrained language models (LMs) are dominated by models that can encode billions of words.
Approach:	They use classifier probing, information-theoretic probing and unsupervised relative acceptability judgments to evaluate model ability.
Outcome:	The proposed models require only about 10M to 100M words to learn to encode most syntactic and semantic features.

Copied to clipboard

Xuhui Sui, Ying Zhang, Kehui Song, Baohang Zhou, Xiaojie Yuan, Wensheng Zhang

Code Summarization with Structure-induced Transformer (2021.findings-acl)

Challenge:	Entity linking is a task of assigning ambiguous mentions in textual input to entities in knowledge bases.
Approach:	They propose a framework to align mentions in text to entities in knowledge bases . they use unsupervised clustering to select key views from descriptions .
Outcome:	The proposed framework achieves state-of-the-art on the zero-shot entity linking dataset.

Copied to clipboard

Hongqiu Wu, Hai Zhao, Min Zhang

Programming Languages in NLP

Challenge:	Code summarization (CS) is a promising area in recent language understanding . previous work using structurebased traversal or non-sequential models to learn structural program semantics has shown no performance gain .
Approach:	They propose to use a structure-based traversal model to learn structural program semantics to generate human language automatically for programming language in the format of source code.
Outcome:	Experiments show that the proposed method achieves state-of-the-art on benchmarks.

Perceptual Structure in the absence of grounding: the impact of abstractedness and subjectivity in color language for LLMs (2023.findings-emnlp)

Copied to clipboard

Pablo Loyola, Edison Marrese-Taylor, Andres Hoyos-Idrobo

Challenge:	Existing studies show that color perception and color language are suitable for empirically studying the problem.
Approach:	They propose to quantify alignment between a defined color space and a feature space in a language model by learning a mapping between embedding space and color space.
Outcome:	The results show that there is considerable alignment between a defined color space and the feature space defined by a language model.

Label Anchored Contrastive Learning for Language Understanding (2022.naacl-main)

Copied to clipboard

Zhenyu Zhang, Yuming Zhao, Meng Chen, Xiaodong He

Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation (2021.eacl-main)

Challenge:	a novel approach to contrastive learning for language understanding is not fully explored . contrastive training has been widely applied to self-supervised representation learning .
Approach:	They propose a label anchored contrastive learning approach for language understanding using a class label.
Outcome:	The proposed approach improves on GLUE and CLUE benchmarks by 4.1% compared to the state-of-the-art approaches . the proposed approach also improves under the few-shot and data imbalance settings .

Copied to clipboard

Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang

Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding (C18-1)

Challenge:	Outdoor vision-and-language navigation (VLN) tasks require visual grounding to generate correct actions.
Approach:	They propose a multimodal text style transfer learning approach to mitigate data scarcity in outdoor vision-and-language navigation tasks.
Outcome:	The proposed approach outperforms baseline models on the outdoor vision-and-language navigation task, improving task completion rate by 8.7% relative to the baseline models.

Copied to clipboard

Yutai Hou, Yijia Liu, Wanxiang Che, Ting Liu

Low-Resource NLP

Challenge:	Existing work which augments an utterance without considering its relation with other utterrances, however, has failed to improve the language understanding module.
Approach:	They propose a sequence-to-sequence generation based data augmentation framework that leverages one utterance’s same semantic alternatives in the training data.
Outcome:	The proposed framework achieves 6.38 and 10.04 F-scores on the Airline Travel Information System dataset and a newly created semantic frame annotation on the Stanford Multi-turn, Multi-domain Dialogue Dataset.

Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models (2021.findings-acl)

Copied to clipboard

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, Ji-Rong Wen

ERNIE: Enhanced Language Representation with Informative Entities (P19-1)

Challenge:	Existing models for KG-to-text generation are based on pretrained language models.
Approach:	They propose to automatically generate a text that describes the facts in knowledge graph (KG) they leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation.
Outcome:	The proposed model outperforms all comparison methods on fully-supervised and fewshot settings.

Copied to clipboard

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu

Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)

Challenge:	Existing pre-trained language models rarely consider incorporating knowledge graphs (KGs) Existing models capture rich semantic patterns from plain text and can be fine-tuned to improve performance of NLP tasks.
Approach:	They propose to incorporate knowledge graphs into pre-trained language models to enhance language representation with external knowledge.
Outcome:	The proposed model can take full advantage of lexical, syntactic, and knowledge information simultaneously.

Copied to clipboard

Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, Kyle Mahowald

Bi-Phone: Modeling Inter Language Phonetic Influences in Text (2023.acl-long)

Challenge:	Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions.
Approach:	They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning.
Outcome:	The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning.

Copied to clipboard

Abhirut Gupta, Ananya B. Sai, Richard Sproat, Yuri Vasilevski, James Ren, Ambarish Jash, Sukhdeep Sodhi, Aravindan Raghuveer

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (2024.acl-long)

Challenge:	Increasingly, people are forced to use the Web in languages they have low literacy in due to technology asymmetries.
Approach:	They propose a method to mine phoneme confusions for pairs of L1 and L2 and plug them into a generative model for synthetically producing corrupted L2 text.
Outcome:	The proposed method corrupts the popular language understanding benchmark SuperGLUE and improves performance.

Copied to clipboard

Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, Kewei Tu

Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)

Challenge:	Existing syntactic language models require a gold tree and sequential training to generate sentences.
Approach:	They propose an unsupervised syntactic language model that incrementally generates a sentence with its syntaktic tree in a left-to-right manner.
Outcome:	The proposed model outperforms existing models on grammar induction and comprehension tasks while holding a substantial acceleration on training.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang

Assessing Composition in Sentence Vector Representations (C18-1)

Challenge:	Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized.
Approach:	They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions.
Outcome:	The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks.

Copied to clipboard

Allyson Ettinger, Ahmed Elgohary, Colin Phillips, Philip Resnik

Representation Learning

Challenge:	opacity of sentence vector representations is a challenge to achieving language understanding . current neural network models are unable to capture meaning information in dense vectors .
Approach:	They propose a method that targets compositional meaning information in sentence embeddings with a high degree of precision and control.
Outcome:	The proposed method extracts useful information about the different capacities of existing sentences models.

An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020.emnlp-main)

Copied to clipboard

Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, Luke Zettlemoyer

Automatic Event Salience Identification (D18-1)

Challenge:	Existing methods to condition models on a concise rationale are less accurate than models that can use the entire context.
Approach:	They propose a method to optimize a bound on the Information Bottleneck objective to extract concise rationales from a binary mask and an end-task predictor that uses only the residual sentences.
Outcome:	The proposed model outperforms existing norm-minimization techniques in task performance and agreement with human rationales in the ERASER benchmark.

Copied to clipboard

Zhengzhong Liu, Chenyan Xiong, Teruko Mitamura, Eduard Hovy

Challenge:	Existing models for analyzing salience of discourse units are inadequate . authors propose two saliency detection models based on discourse relations .
Approach:	They propose two salience detection models based on discourse relations that capture complex interactions between discourse units.
Outcome:	The proposed models outperform the strong frequency baseline and improve the feature based model by a large margin.

DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension (P18-1)

Copied to clipboard

Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, Karthik Sankaranarayanan

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information (2021.acl-long)

Challenge:	DuoRC contains 186,089 unique question-answer pairs created from 7680 movie plots .
Approach:	They propose a novel dataset for Reading Comprehension that motivates new challenges for neural approaches in language understanding beyond those offered by existing RC datasets.
Outcome:	The proposed dataset motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets.

Copied to clipboard

Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, Jiwei Li

Dependency Parsing for Spoken Dialog Systems (D19-1)

Challenge:	ChineseBERT model incorporates glyph and pinyin information of Chinese characters into pretraining . proposed model achieves new performance boost over baseline models with fewer training steps .
Approach:	They propose a ChineseBERT model that incorporates glyph and pinyin information into pretraining . the glyph embedding is obtained based on different fonts of a character, and the pinyink embeddment characterizes the pronunciation of Chinese characters.
Outcome:	The proposed model achieves new performance boosts over baseline models with fewer training steps.

Copied to clipboard

Sam Davidson, Dian Yu, Zhou Yu

Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoning (2024.naacl-long)

Challenge:	Dependency parsing of conversational input can help to understand dialogs . currently available annotation schemes do not adapt well to spoken human-machine dialogs.
Approach:	They propose an annotation scheme that extends Universal Dependencies guidelines to spoken dialogs.
Outcome:	The proposed scheme disambiguates relationships between entities extracted from dialogs . it is better than existing models on public datasets and fine-tuned on ConvBank data .

Copied to clipboard

Athul Jacob, Gabriele Farina, Jacob Andreas

WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines (2025.naacl-long)

Challenge:	Existing computational models of pragmatics are implemented as recursive reasoning procedures, in which listeners interpret utterances by reasoning about the intentions of less-sophisticated speakers.
Approach:	They propose a game-theoretic model of pragmatics that combines a recursive reasoning model with a regularized conventions model to obtain principled pragmatic language generation procedures.
Outcome:	The proposed model matches or improves predictions made by iterated best response and Rational Speech Acts models of language understanding.

Copied to clipboard

Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, Chong-Wah Ngo

BasqueGLUE: A Natural Language Understanding Benchmark for Basque (2022.lrec-1)

Challenge:	Vision Language Models struggle with cultural-specific knowledge, especially in languages other than English and in underrepresented cultural contexts.
Approach:	They propose a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects and a training dataset.
Outcome:	The proposed model performs better with correct location context, but struggles with adversarial contexts and predicting specific regional cuisines and languages.

Copied to clipboard

Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, Aitor Soroa

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants (N18-1)

Challenge:	Natural Language Understanding (NLU) benchmarks are costly to develop and language-dependent . basqueGLUE is the first benchmark for Basque, a less-resourced language .
Approach:	They propose a benchmark for Basque, a less-resourced language, using existing datasets.
Outcome:	The proposed benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding beyond the detection of superficial clues.

Copied to clipboard

Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein

Eliciting Implicit Acoustic Styles from Open-domain Instructions to Facilitate Fine-grained Controllable Generation of Speech (2025.emnlp-main)

Challenge:	Existing methods for analyzing warrants in natural language arguments are insufficient.
Approach:	They propose a method for reconstructing warrants from news comments . they use a crowdsourcing process to obtain warrants for 2k authentic arguments .
Outcome:	The proposed method will define a substantial step towards automatic warrant reconstruction.

Copied to clipboard

Jianxing Yu, Gou Zihao, Chen Li, Zhisheng Wang, Peiji Yang, Wenqing Chen, Jian Yin

Enhancing the General Agent Capabilities of Low-Paramter LLMs through Tuning and Multi-Branch Reasoning (2024.findings-naacl)

Challenge:	Current work relies on pre-defined rules or templates to control the style of speech.
Approach:	They propose to use open-domain instructions to generate speech with the acoustic style that meets users’ needs based on their instructions.
Outcome:	The proposed model can be used to generate speech with the acoustic style that meets users’ needs based on open-domain instructions.

Copied to clipboard

Qinhao Zhou, Zihan Zhang, Xiang Xiang, Ke Wang, Yuchuan Wu, Yongbin Li

A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation (2022.findings-acl)

Challenge:	Open-source pre-trained Large Language Models exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks.
Approach:	They propose a method to construct agent-specific data using GPT-4 and supervised fine-tuning . they find that supervised tunning can significantly reduce hallucination outputs and formatting errors in agent tasks .
Outcome:	The proposed method improves on five agent tasks of AgentBench.

Copied to clipboard

Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, Xipeng Qiu

Challenge:	Existing methods to measure instance difficulty use generalization and threshold-tuning . a new approach to learn to exit is based on hash functions to assign tokens to a fixed exiting layer.
Approach:	They propose a Hash-based Early Exiting approach that replaces learn-to-exit modules with hash functions to assign each token to a fixed exiting layer.
Outcome:	The proposed approach improves on learning to exit and predicting instance difficulty.

Robustness Testing of Language Understanding in Task-Oriented Dialog (2021.acl-long)

Copied to clipboard

Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, Minlie Huang

F2TEval: Human-Aligned Multi-Dimensional Evaluation for Figure-to-Text Task (2025.emnlp-main)

Challenge:	a lack of systematic studies on the robustness of language understanding models in task-oriented dialog systems is limiting . authors propose a model-agnostic toolkit LAUG to approximate natural language perturbations .
Approach:	They propose a model-agnostic toolkit LAUG to approximate natural language perturbations for testing the robustness of language understanding models in task-oriented dialog systems.
Outcome:	The proposed toolkit reveals critical robustness issues in state-of-the-art models.

Copied to clipboard

Tan Yue, Rui Mao, Zilong Song, Zonghai Hu, Dongyan Zhao

Green & Sustainable NLP

Challenge:	Existing evaluation methods for Figure-to-Text tasks are limited due to the inherent ambiguity and semantic compression of figures, the generated texts suffer from factual inaccuracies, incomplete coverage, and weak logical reasoning.
Approach:	They propose a five-dimensional reference-free evaluation method aligned with expert criteria to support fine-grained evaluation.
Outcome:	The proposed method outperforms Gemini-2.0 and Claude-3.5 with only 0.9B parameters.

Unified Pre-training for Program Understanding and Generation (2021.naacl-main)

Copied to clipboard

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Text Generation

Challenge:	PLUG is a programming language that is used for programming and language understanding and generation tasks.
Approach:	They propose a sequence-to-sequence model that performs a broad spectrum of program and language understanding and generation tasks.
Outcome:	The proposed model outperforms or rivals state-of-the-art models on code summarization, code generation, and code translation tasks in seven programming languages.

Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism (2024.emnlp-main)

Copied to clipboard

Lang Cao

Probing Pre-trained Language Models for Semantic Attributes and their Values (2021.findings-emnlp)

Challenge:	Recent advances in large language models have demonstrated impressive language understanding and generation capabilities, enabling them to answer a wide range of questions across various domains.
Approach:	They propose a refusal mechanism that instructs LLMs to refuse to answer challenging questions in order to avoid errors.
Outcome:	The proposed approach improves the controllability and reliability of large language models and their ability to answer questions across domains.

Copied to clipboard

Meriem Beloucif, Chris Biemann

A fine-grained comparison of pragmatic language understanding in humans and language models (2023.acl-long)

Challenge:	Pretrained language models (PTLMs) are used for many tasks including syntax, semantics and commonsense.
Approach:	They propose to integrate semantic attributes and their values into pretrained language models to improve their performance on many natural language processing tasks.
Outcome:	The proposed model performs better on masked tokens than humans on this task.

Copied to clipboard

Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, Edward Gibson

PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction (2021.acl-long)

Challenge:	Pragmatics and non-literal language understanding are essential to human communication . a long-standing challenge for artificial language models is to capture pragmatics .
Approach:	They compare language models and humans on seven pragmatic phenomena using curated English materials.
Outcome:	The proposed model achieves high accuracy and matches human error patterns . the results suggest pragmatic behaviors can emerge in models without explicit representations of mental states .

Copied to clipboard

Shulin Liu, Tao Yang, Tianchi Yue, Feng Zhang, Di Wang

A Dataset for N-ary Relation Extraction of Drug Combinations (2022.naacl-main)

Challenge:	Chinese spelling correction (CSC) is a task to detect and correct spelling errors in texts.
Approach:	They propose a Pre-trained masked Language model with Misspelled knowledgE (PLOME) which jointly learns how to understand language and correct spelling errors.
Outcome:	The proposed model outperforms state-of-the-art methods on widely used benchmarks and achieves superior performance against existing models.

Copied to clipboard

Aryeh Tiktinsky, Vijay Viswanathan, Danna Niezni, Dana Meron Azagury, Yosi Shamay, Hillel Taub-Tabib, Tom Hope, Yoav Goldberg

Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022.naacl-main)

Challenge:	Combination therapies are becoming standard of care for diseases such as cancer, tuberculosis, malaria and HIV.
Approach:	They construct an expert-annotated dataset for extracting drug combinations from the scientific literature.
Outcome:	The proposed dataset is the first relation extraction dataset consisting of variable-length relations.

Copied to clipboard

Zeming Chen, Qiyue Gao

Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study (2025.findings-acl)

Challenge:	Existing evaluation methods do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning.
Approach:	They propose a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena using a set of datasets and an evaluation procedure for diagnosing how well a language model captures reasoning skills.
Outcome:	The proposed model can diagnose model behavior and verify model learning quality.

Copied to clipboard

Pablo Rodríguez, Silvia Paniagua Suárez, Pablo Gamallo, Susana Sotelo Docio

Extract, Integrate, Compete: Towards Verification Style Reading Comprehension (2021.findings-emnlp)

Challenge:	Recent advances in large language models have led to remarkable improvements in language understanding and text generation.
Approach:	They propose a framework to evaluate large language models for underrepresented languages . they examine CPT strategies for languages with limited representation in multilingual models .
Outcome:	The proposed evaluation framework is based on the case of Galician language . it assesses trade-offs between linguistic enrichment and task-solving capabilities .

Copied to clipboard

Chen Zhang, Yuxuan Lai, Yansong Feng, Dongyan Zhao

MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)

Challenge:	VGaokao is a verification style reading comprehension dataset for Chinese language tests requiring advanced language understanding skills.
Approach:	They propose a new extract-integration-compete approach to extract complementary evidence from Chinese Language tests of Gaokao and a pairwise competition to push models to learn the subtle difference between similar text pieces.
Outcome:	The proposed approach outperforms baselines on VGaokao with retrieved complementary evidence while having the merits of efficiency and explainability.

Copied to clipboard

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

Leveraging Grammar Induction for Language Understanding and Generation (2024.findings-emnlp)

Challenge:	Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation.
Approach:	They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.
Outcome:	The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models.

Copied to clipboard

Jushi Kai, Shengyuan Hou, Yusheng Huang, Zhouhan Lin

Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding (2026.acl-long)

Challenge:	Existing grammar induction methods do not provide sufficient performance in downstream tasks.
Approach:	They propose an unsupervised grammar induction method for language understanding and generation using a grammar parser and a syntactic mask.
Outcome:	The proposed method performs better on from-scratch and pre-trained scenarios.

Copied to clipboard

Beomsik Cho, Jaehyung Kim

Reference-free Hallucination Detection for Large Vision-Language Models (2024.findings-emnlp)

Challenge:	Large Vision–Language Models (LVLMs) integrate visual perception with language understanding, but how vision information contributes to the model’s decoding process remains under-explored.
Approach:	They propose a simple training-free decoding method that guides text generation in Large Vision–Language Models by Referencing Vision Tokens.
Outcome:	The proposed method leverages the semantic information embedded within vision tokens by projecting it into the text token distribution.

Copied to clipboard

Qing Li, Jiahui Geng, Chenyang Lyu, Derui Zhu, Maxim Panov, Fakhri Karray

In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax (2024.naacl-long)

Challenge:	Large vision-language models exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, but they are prone to producing hallucinations.
Approach:	They propose to use supervised uncertainty quantification methods to detect hallucinations in large vision-language models.
Outcome:	The proposed methods outperform the others in detecting hallucinations on four representative LVLMs across two different tasks.

Copied to clipboard

Aaron Mueller, Albert Webson, Jackson Petty, Tal Linzen

Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)

Challenge:	In-context learning is a common method for teaching large language models new tasks . given labeled examples in the input context, the model learns to perform the task without weight updates.
Approach:	They examine whether models guided via ICL infer the underlying structure of the task defined by the context or rely on superficial heuristics that only generalize to identically distributed examples.
Outcome:	The proposed model generalizes syntactically or linearly on out-of-distribution examples . the proposed model is able to generalize better on pre-trained models .

Copied to clipboard

Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang

LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models (2023.findings-acl)

Challenge:	Z-Code++ is a pre-trained language model optimized for abstractive text summarization.
Approach:	They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance.
Outcome:	The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings.

Copied to clipboard

Philipp Wicke

Korean Language Modeling via Syntactic Guide (2022.lrec-1)

Challenge:	Figures are based on the use of words in a way that deviates from their conventional order and meaning.
Approach:	They propose to use a figurative language model to interpret embodied metaphors by using larger language models that conceptualise embodies the action of the metaphorical sentence.
Outcome:	The proposed model enables interpretation of figurative language when the action of the metaphorical sentence is more embodied.

Copied to clipboard

Hyeondey Kim, Seonhoon Kim, Inho Kang, Nojun Kwak, Pascale Fung

Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets (D18-1)

Challenge:	Existing research on pre-trained language models focuses on widely-used languages . however, not every language can benefit from such models due to computational resources .
Approach:	They propose to build a pre-trained language model that understands the linguistic phenomena in the target language with low resources.
Outcome:	The proposed model improves the performance of Korean language understanding tasks.

Copied to clipboard

Nathan Greenberg, Trapit Bansal, Patrick Verga, Andrew McCallum

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation (2021.acl-long)

Challenge:	Existing large labeled text datasets contain labels for multiple subsets of biomedical entity types, but it is rare to find large labeling datasets containing all desired entity types together.
Approach:	They propose a method for training a single CRF extractor from multiple datasets with disjoint or partially overlapping sets of entity types.
Outcome:	The proposed method improves NER F1 over training in isolation on biocreative V CDR, biocreativ VI ChemProt and MedMentions datasets.

Copied to clipboard

Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, Luo Si

Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension? (2021.eacl-main)

Challenge:	Existing work in multilingual pretraining relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages.
Approach:	They propose to plug a cross-attention module into a Transformer encoder to explicitly build the interdependence between languages.
Outcome:	The proposed model outperforms existing models on XTREME and English-to-French translation datasets.

Copied to clipboard

Yulong Wu, Viktor Schlegel, Riza Batista-Navarro

What do we expect from Multiple-choice QA Systems? (2020.findings-emnlp)

Challenge:	Existing benchmarks for machine reading comprehension (MRC) are insufficient to assess models for their capabilities to read and comprehend .
Approach:	They propose an ablation-based method to assess the extent to which MRC datasets evaluate the understanding of explicit discourse relations.
Outcome:	The proposed method shows that the model's performance drops on three large-scale datasets . the results suggest that most of the answers do not require understanding the discourse structure of the text.

Copied to clipboard

Krunal Shah, Nitish Gupta, Dan Roth

Question Answering

Challenge:	Recent work has shown that good performance on a dataset might not correlate well with human’s expectations from models that “understand” language.
Approach:	They propose to train a top performing multiple choice question answering model against expectations from models that "understand" language.
Outcome:	The proposed training paradigm leads to a model that performs on par with the original model while better satisfying our expectations.

Exploring Numeracy in Word Embeddings (P19-1)

Copied to clipboard

Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, Eduard Hovy

Representation Learning

Challenge:	Existing word embeddings are inadequate at capturing numerical properties of numbers.
Approach:	They propose to use word embeddings to capture numerical properties of numbers . they hope to develop methods which better capture numeric properties .
Outcome:	The proposed models lack the ability to capture numeric properties of numbers, the authors show . their findings provide a starting point for the development of better models .

Zero-Shot Entity Linking by Reading Entity Descriptions (P19-1)

Copied to clipboard

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, Honglak Lee

Modelling Commonsense Properties Using Pre-Trained Bi-Encoders (2022.coling-1)

Challenge:	Existing approaches to link entities to unseen entities require in-domain labeled data.
Approach:	They propose a zero-shot entity linking task where mentions must be linked to unseen entities without in-domain labeled data.
Outcome:	The proposed task can generalize to unseen entities without metadata or alias tables . the proposed system improves over baselines, including BERT, on a new dataset .

Copied to clipboard

Amit Gajbhiye, Luis Espinosa-Anke, Steven Schockaert

ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding (2025.findings-naacl)

Challenge:	Pre-trained language models can capture commonsense properties that are rarely expressed in text.
Approach:	They propose to fine-tune language models to explicitly model commonsense properties . they train separate concept and property encoders on extracted hyponym-hypernym pairs and generic sentences .
Outcome:	The proposed model can capture commonsense properties with higher accuracy than human models . a new study shows that the model can model commonsensence properties with much higher accuracy .

Copied to clipboard

Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha Tesfu, Philipp Slusallek, Thamar Solorio, Dietrich Klakow

Visually Grounded Compound PCFGs (2020.emnlp-main)

Challenge:	Large language models (LLMs) evaluation is gaining increasing attention as they are typically trained on general-domain datasets while demonstrating notable performance on tasks out of their training domains.
Approach:	They propose an LLM evaluation benchmark for low-resource languages that focuses on low-rsource language understanding in culture-specific scenarios.
Outcome:	The proposed benchmarks outperform monolingual evaluations on proverb generation tasks and native language proverb descriptions on multiple choice tasks.

Copied to clipboard

Yanpeng Zhao, Ivan Titov

SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoc (2025.naacl-long)

Challenge:	Existing work on visual groundings for language understanding has been drawing much attention.
Approach:	They propose to use an extension of probabilistic context-free grammar model to do fully-differentiable end-to-end visually grounded learning.
Outcome:	The proposed model outperforms the previous grounded model and significantly outperformed the previous model on the MSCOCO test captions.

Copied to clipboard

Daniel Guzman Olivares, Lara Quijano, Federico Liberatore

IMPLI: Investigating NLI Models’ Performance on Figurative Language (2022.acl-long)

Challenge:	Recent studies show that language understanding offered by chat-based Large Language Models is limited and far from human-like performance.
Approach:	They propose a framework for multilingual Word Sense Disambiguation using group algebra.
Outcome:	The proposed framework surpasses the performance of current alternatives even in low-resource languages while reducing the parameter count by 72%.

Copied to clipboard

Kevin Stowe, Prasetya Utama, Iryna Gurevych

Sentiment Analysis

Challenge:	Understanding figurative language is a difficult area in NLP but is essential for proper understanding.
Approach:	They propose to use a dataset to generate 24k semiautomatic pairs and manually create 1.8k gold pairs to evaluate NLI models.
Outcome:	The proposed models can detect entailment relationship between figurative phrases and their literal counterparts, but perform poorly on similar structured examples.

Systematic Inequalities in Language Technology Performance across the World’s Languages (2022.acl-long)

Copied to clipboard

Damian Blasi, Antonios Anastasopoulos, Graham Neubig

Challenge:	Recent studies have revealed that NLP is limited to a subset of the world’s 6,500 languages.
Approach:	They propose a framework for estimating the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP.
Outcome:	The proposed framework estimates the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP.

Revealing the Parallel Multilingual Learning within Large Language Models (2024.emnlp-main)

Copied to clipboard

Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, JingBo Zhu

Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding (2023.emnlp-main)

Challenge:	Large language models (LLMs) can handle multilingual and cross-lingual text within a single input; however, previous studies focusing on using English as the pivot language to enhance language understanding and reasoning focus on using multiple languages.
Approach:	They propose to use parallel multilingual input to enhance the model's comprehension of the input and to examine how multilingual processing affects prediction.
Outcome:	The proposed model can handle multilingual and cross-lingual text within a single input, but previous studies focused on using English as the pivot language to enhance language understanding and reasoning.

Copied to clipboard

Shuwen Deng, Paul Prasse, David Reich, Tobias Scheffer, Lena Jäger

MockingBERT: A Method for Retroactively Adding Resilience to NLP Models (2022.coling-1)

Challenge:	Existing models for augmenting language models with human scanpaths have been developed, but the potential of synthetic gaze data across NLP tasks remains unexplored.
Approach:	They propose to combine synthetic scanpath generation with a scanpath-augmented language model, eliminating the need for human gaze data.
Outcome:	The proposed model outperforms the underlying language model and achieves comparable performance to a language model augmented with real human gaze data.

Copied to clipboard

Jan Jezabek, Akash Singh

Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding (2021.findings-emnlp)

Challenge:	Existing remediations have compromised accuracy or required full model re-training with each new class of attacks.
Approach:	They propose a method of retroactively adding resilience to misspellings to transformer-based NLP models and propose generating adversarial misspells using an approximate method.
Outcome:	The proposed method significantly reduces the cost needed to evaluate a model’s resilience to adversarial attacks.

Copied to clipboard

Shane Storks, Qiaozi Gao, Yichi Zhang, Joyce Chai

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (D19-1)

Challenge:	Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks.
Approach:	They propose a commonsense reasoning dataset with dense annotations that allows multi-tiered evaluation of machines’ reasoning process.
Outcome:	The proposed model can achieve high end performance but struggle to support predictions with valid supporting evidence.

Copied to clipboard

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, Ruslan Salakhutdinov

Joint Intent Detection and Entity Linking on Spatial Domain Queries (2020.findings-emnlp)

Challenge:	Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction.
Approach:	They propose a new formulation of attention via the lens of the kernel which allows us to understand individual components of Transformer's attention.
Outcome:	The proposed model outperforms existing models on language understanding and sequence prediction tasks and is more efficient than existing models.

Copied to clipboard

Lei Zhang, Runze Wang, Jingbo Zhou, Jingsong Yu, Zhenhua Ling, Hui Xiong

Sentiment Analysis

Challenge:	Spatial domain queries have unique properties making them more challenging for language understanding than common conversational queries.
Approach:	They propose a language understanding framework for spatial domain queries that jointly learns the intent detection and entity linking tasks on a voice assistant service.
Outcome:	The proposed framework outperforms baseline methods with a significant margin.

H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables (2025.naacl-long)

Copied to clipboard

Nikhil Abhyankar, Vivek Gupta, Dan Roth, Chandan K. Reddy

Exploiting Commonsense Knowledge about Objects for Visual Activity Recognition (2023.findings-acl)

Challenge:	Existing methods for tabular reasoning combine textual and symbolic reasoning in a two-stage process to address these limitations.
Approach:	They propose an algorithm that integrates symbolic and semantic (textual) approaches in a two-stage process to address these limitations.
Outcome:	The proposed algorithm outperforms state-of-the-art methods across three tabular question-answering and fact-verification datasets, underscoring its effectiveness and efficiency.

Copied to clipboard

Tianyu Jiang, Ellen Riloff

SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining (2021.acl-long)

Challenge:	Existing tasks that aim to identify the objects in an image are object detection and image classification, but recent work has focused on more comprehensive image under- standing tasks.
Approach:	They propose to incorporate commonsense knowledge about physical objects into a transformer-based model that is trained to predict the actionverb for visual activity recognition.
Outcome:	The proposed model incorporates prototypical function knowledge about physical objects to predict the actionverb for visual activity recognition.

Copied to clipboard

Taolin Zhang, Zerui Cai, Chengyu Wang, Minghui Qiu, Bite Yang, Xiaofeng He

LLM-OREF: An Open Relation Extraction Framework Based on Large Language Models (2025.emnlp-main)

Challenge:	Existing knowledge-based PLMs are based on linked-entity information, but they only use linked-enemy information as auxiliary information.
Approach:	They propose to integrate semantic knowledge from neighbours of linked-entity into a medical PLM that integrates heterogeneous-entities into the homogeneously neighbouring entity structure.
Outcome:	Experiments show that SMedBERT outperforms baselines in knowledge-intensive Chinese medical tasks.

Copied to clipboard

Hongyao Tu, Liang Zhang, Yujie Lin, Xin Lin, Haibo Zhang, Long Zhang, Jinsong Su

ParaMac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms (2022.findings-emnlp)

Challenge:	Existing studies focus on building models that can only handle predefined relations . however, their reliance on human annotation limits their practicality .
Approach:	They propose an open relation extraction framework that can generalize to new relations not encountered during training.
Outcome:	The proposed framework can generalize to new relations not encountered during training.

Copied to clipboard

Jinxin Liu, Jiaxin Shi, Ji Qi, Lei Hou, Juanzi Li, Qi Tian

Text Generation

Challenge:	Existing unsupervised methods for paraphrase generation are weak in semantic equivalence or expression diversity.
Approach:	They propose a framework for unsupervised paraphrase generation that employs multi-aspect equivalence constraints and multi-granularity diversifying mechanisms to achieve good semantic equvalence and expressive diversity.
Outcome:	The proposed framework achieves 9.1% and 3.3% absolute gains over previous SOTA on Quora and MSCOCO and can improve to 18.0% and 4.6% on GLUE.

Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)

Copied to clipboard

Amilleah Rodriguez, Shaonan Wang, Liina Pylkkänen

Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation (2022.emnlp-main)

Challenge:	a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure.
Approach:	They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions.
Outcome:	The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses .

Copied to clipboard

Wenhao Wu, Wei Li, Jiachen Liu, Xinyan Xiao, Sujian Li, Yajuan Lyu

AlgoPuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Algorithmic Multimodal Puzzles (2025.naacl-long)

Challenge:	Existing models for text generation are weak enough to handle perturbations in inputs, leading to degeneration in faithfulness and informativeness.
Approach:	They propose a framework for improving faithfulness and informativeness of Seq2Seq models by perturbing word representations and word swapping.
Outcome:	The proposed framework improves faithfulness and informativeness of Seq2Seq models under automatic and human evaluation settings.

Copied to clipboard

Deepanway Ghosal, Vernon Toh, Yew Ken Chia, Soujanya Poria

DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation (2023.acl-long)

Challenge:	Existing datasets focused on visual question-answering focus on visual, language, and algorithmic knowledge . a new study examines the performance of multimodal language models in solving algorithmic puzzles .
Approach:	They propose a dataset to test the capabilities of multimodal language models in solving algorithmic puzzles.
Outcome:	The proposed dataset is generated automatically from human code.

Copied to clipboard

Yuxi Feng, Xiaoyuan Yi, Xiting Wang, Laks Lakshmanan, V.S., Xing Xie

Revisiting Demonstration Selection Strategies in In-Context Learning (2024.acl-long)

Challenge:	Existing approaches to augment self-training (ST) in attribute-controllable language generation are limited and limited.
Approach:	They propose a new ST framework that integrates self-generated pseudo text into attribute-controllable language generation.
Outcome:	The proposed framework can be applied to semi-supervised controllable language generation.

Copied to clipboard

Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao

Low-Resource NLP

Challenge:	Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL).
Approach:	They propose a data- and model-dependent method to select models using in-context learning, TopK + ConE, and propose unified explanations for the effectiveness of previous methods.
Outcome:	The proposed method improves language understanding and generation tasks with different model scales.

Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean (2025.acl-long)

Copied to clipboard

SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee

Challenge:	Existing studies have focused on linguistic competence of language models with grammatical knowledge.
Approach:	They propose to use grammar as a measurable proxy to assess linguistic competence of large language models (LLMs) .
Outcome:	The proposed model aims to assess the linguistic competence of large language models (LLMs) and humans in Korean.

How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)

Copied to clipboard

Lovisa Hagström, Richard Johansson

POSQA: Probe the World Models of LLMs with Size Comparisons (2023.findings-emnlp)

Challenge:	Current language models have been criticised for learning language from text alone without connection between words and their meaning.
Approach:	They propose to train models on more sources than text to provide the lacking connection between words and their meanings.
Outcome:	The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation.

Copied to clipboard

Chang Shu, Jiuzhou Han, Fangyu Liu, Ehsan Shareghi, Nigel Collier

MusKGC: A Flexible Multi-source Knowledge Enhancement Framework for Open-World Knowledge Graph Completion (2025.emnlp-main)

Challenge:	Embodied language comprehension emphasizes that language understanding is not only mental processing in the brain but also involves interactions with the physical and social environment.
Approach:	They propose to use a physical object size question to examine the extremity of large language models to test their embodied comprehension.
Outcome:	The proposed dataset shows that even the largest LLMs perform poorly under the zero-shot setting.

Copied to clipboard

Xin Song, Liu Haiyan, Haiyang Wang, Ye Wang, Kai Chen, Bin Zhou

Structured Data in NLP

Challenge:	Open-world knowledge graph completion (KGC) aims to infer novel facts by enriching existing graphs with external knowledge sources while maintaining semantic consistency under the open-world assumption (OWA).
Approach:	They propose a multi-source knowledge enhancement framework based on an open-world assumption (OWA) that integrates external knowledge sources and a new evaluation strategy to validate new facts.
Outcome:	The proposed model achieves SOTA performance across benchmarks and the evaluation strategy effectively assesses new facts under OWA.

Culturally Aware Natural Language Inference (2023.findings-emnlp)

Copied to clipboard

Jing Huang, Diyi Yang

Open Domain Web Keyphrase Extraction Beyond Language Modeling (D19-1)

Challenge:	Cultural norms are behavioral rules and conventions shared within specific groups, connecting cultural symbols and values.
Approach:	They propose a task that operationalizes cultural variations in language understanding through a natural language inference task that surfaces cultural variations as label disagreement between annotators from different cultural groups.
Outcome:	The proposed model can be evaluated at which levels it is culturally aware.

Copied to clipboard

Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, Arnold Overwijk

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation (2024.findings-acl)

Challenge:	Recent neural methods for keyphrase extraction are mostly observed in documents originating from the scientific domain.
Approach:	They develop a neural keyphrase extraction model that goes beyond language understanding to handle the variations of domain and content quality.
Outcome:	The proposed model can handle the variations of domain and content quality without restriction of the domain, quality, nor content of the documents.

Copied to clipboard

Yi Zong, Xipeng Qiu

SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation (2025.naacl-long)

Challenge:	Existing datasets focus on primary perception abilities and commonsense knowledge, or have low level of text comprehension difficulty, which are insufficient to reflect comprehensive capabilities of large vision-language models.
Approach:	They propose a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO) which sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning.
Outcome:	The proposed model derives from native Chinese context and sets human-level requirements for its abilities, including perception, understanding, knowledge and reasoning.

Copied to clipboard

Yixian Shen, Qi Bi, Jia-hong Huang, Hongyi Zhu, Andy D. Pimentel, Anuj Pathania

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023.acl-long)

Challenge:	Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM).
Approach:	They propose a low-rank adaptation approach that reduces the number of trainable parameters while enhancing model performance.
Outcome:	The proposed approach outperforms existing parameter-efficient fine-tuning methods while achieving substantial reductions in computational cost and memory requirements.

Copied to clipboard

Jian Yang, Shuming Ma, Li Dong, Shaohan Huang, Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, Zhoujun Li

SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge (2020.emnlp-main)

Challenge:	Existing pre-training methods underutilize the benefits of language understanding for generation.
Approach:	They propose a GAN-style model for encoder-decoder pre-training with an auxiliary discriminator.
Outcome:	The proposed model outperforms existing pre-trained models and achieves state-of-the-art performance.

Copied to clipboard

Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, Minlie Huang

Multilingual Multi-Figurative Language Detection (2023.findings-acl)

Challenge:	Existing pre-trained models neglect to consider linguistic knowledge of texts . existing models neglect linguistic information, which is important for sentiment analysis .
Approach:	They propose a model that introduces word-level linguistic knowledge into pre-trained models to enhance sentiment analysis by querying SentiWordNet to acquire sentiment polarity.
Outcome:	The proposed model obtains state-of-the-art performance on a variety of sentiment analysis tasks.

Copied to clipboard

Huiyuan Lai, Antonio Toral, Malvina Nissim

On General Language Understanding (2023.findings-emnlp)

Challenge:	Figures of speech help people express abstract concepts and emotions, but it's understudied in a multilingual setting and when considering more than one figure of speech at the same time.
Approach:	They propose a framework for sentence-level figurative language detection based on template-based prompt learning and use it to unify multiple detection tasks that are interrelated across multiple figures of speech and languages.
Outcome:	The proposed framework outperforms baselines and may serve as blueprint for the joint modelling of other interrelated tasks.

Copied to clipboard

David Schlangen

Challenge:	a recent paper suggests that the evidence underspecifies the understanding of large language models.
Approach:	They propose to use a "general language understanding" benchmark to examine what it could mean in machines.
Outcome:	The proposed model can be used to ground questions of the adequacy of benchmarking methods.

Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant (2025.findings-emnlp)

Copied to clipboard

Lei Shen, Xiaoyu Shen

Challenge:	Auto-SLURP is a benchmark dataset for evaluating multi-agent frameworks powered by large language models.
Approach:	Auto-SLURP is a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks . authors propose it extends original SLURP dataset by relabeling data and integrating simulated servers and external services.
Outcome:	The proposed dataset extends the original SLURP dataset for natural language understanding tasks.

CWID-hi: A Dataset for Complex Word Identification in Hindi Text (2022.lrec-1)

Copied to clipboard

Gayatri Venugopal, Dhanya Pramod, Ravi Shekhar

Information Retrieval

Challenge:	Text simplification is a method for improving the accessibility of text by converting complex sentences into simple sentences.
Approach:	They propose to use Hindi knowledge annotators to capture the annotator’s language knowledge to build an automatic complex word classifier using a soft voting approach.
Outcome:	The proposed dataset shows that native and non-native annotators perceive complex words differently depending on their language knowledge.

Can You Unpack That? Learning to Rewrite Questions-in-Context (D19-1)

Copied to clipboard

Ahmed Elgohary, Denis Peskov, Jordan Boyd-Graber

Are representations built from the ground up? An empirical examination of local composition in language models (2022.emnlp-main)

Challenge:	Existing QA datasets lack key NLP problems like coreference and ellipsis resolution.
Approach:	They propose a task of question-in-context rewriting to rewrite a context-dependent question into a self-contained question with the same answer.
Outcome:	The proposed task is based on a dataset of 40,527 questions based in QuAC . it requires models to link questions together to resolve conversational dependencies .

Copied to clipboard

Emmy Liu, Graham Neubig

Strategy-level Entrainment of Dialogue System Users in a Creative Visual Reference Resolution Task (2022.lrec-1)

Challenge:	Compositionality is a hallmark of human language, but many phrases are non-compositional . a study by a team of researchers shows that LMs may not be able to distinguish between compositional and non-composable phrases.
Approach:	They propose to predict LM-internal representations of longer phrases given their constituents . they find that the representation of a parent phrase can be predicted with some accuracy .
Outcome:	The proposed model can predict a parent phrase with some accuracy given its children's transformations, but this is not the case.

Copied to clipboard

Deepthi Karkada, Ramesh Manuvinakurike, Maike Paetzel-Prüsmann, Kallirroi Georgila

Comparing Approaches to Language Understanding for Human-Robot Dialogue: An Error Taxonomy and Analysis (2022.lrec-1)

Challenge:	entrainment is a phenomenon in which interlocutors start speaking more similarly to each other.
Approach:	They propose to use crowd-sourced data to study entrainment of users playing a creative reference resolution game with an autonomous dialogue system.
Outcome:	The proposed system adapts the user's descriptive strategy to one that is simpler to parse for the natural language understanding unit without impinging on their creativity.

Copied to clipboard

Ada Tur, David Traum

Program Enhanced Fact Verification with Verbalization and Graph Attention Network (2020.emnlp-main)

Challenge:	Existing approaches to language understanding for human-robot interaction are limited by domain-specific grammars and domain-level inputs.
Approach:	They compare a relevance-based classifier with a GPT-2 model and compare their results . they find that the relevance- and GPT-based models make different errors .
Outcome:	The proposed model outperforms the existing model with 2000 examples as training data.

Copied to clipboard

Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, Xiaodan Zhu

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024.emnlp-main)

Challenge:	Existing methods for fact verification based on structured data are challenging and require further study.
Approach:	They propose a program-enhanced verbalization and a graph attention network to integrate programs and execution into textual inference models.
Outcome:	The proposed framework achieves a new state-of-the-art accuracy on a benchmark dataset . it is compared with existing frameworks on symbolic and informal inference models .

Copied to clipboard

Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, Iryna Gurevych

ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022.emnlp-main)

Challenge:	Recent prompting techniques have improved LLMs’ performance on various reasoning tasks, but there is little understanding of what triggers reasoning abilities in LLM in the inference stage.
Approach:	They propose a method that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution.
Outcome:	The proposed method boosts multiple LLMs by 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral.

Copied to clipboard

Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, Dilek Hakkani-Tur

Unsupervised Entity Linking with Guided Summarization and Multiple-Choice Selection (2022.emnlp-main)

Challenge:	Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments.
Approach:	They propose to use ALFRED to test whether models can adapt to tasks not seen during training that require the same types of language understanding as ALFred.
Outcome:	The proposed model can adapt to tasks that require the same types of language understanding as ALFRED-L.

Copied to clipboard

Young Min Cho, Li Zhang, Chris Callison-Burch

Introducing Semantics into Speech Encoders (2023.acl-long)

Challenge:	Entity linking is an important task for language understanding.
Approach:	They propose a fully unsupervised model that generates a guided summary of the contexts conditioning on a mention and then casts the task to a multiple-choice problem.
Outcome:	The proposed model achieves state-of-the-art performance on existing datasets and exiting datasets.

Copied to clipboard

Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang

New Protocols and Negative Results for Textual Entailment Data Collection (2020.emnlp-main)

Challenge:	Existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Approach:	They propose a task-agnostic unsupervised way to incorporate semantic information from large language model (LLM) systems into self-supervised speech encoders without labeled audio transcriptions.
Outcome:	The proposed approach improves spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) score by over 22%.

Copied to clipboard

Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares, Emily Pitler

Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels? (2024.findings-emnlp)

Challenge:	Natural language inference data has proven useful in benchmarking and as pretraining data for tasks requiring language understanding.
Approach:	They propose four alternative protocols to improve annotation quality and diversity . they use 8.5k-example training sets to compare different protocols .
Outcome:	The proposed protocols improve the ease of training and quality of the examples.

Copied to clipboard

Yidan Zhang, Zhenan He

What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)

Challenge:	Large language models (LLMs) still exhibit significant deficiencies in basic language understanding and manipulation.
Approach:	They propose a bilingual benchmark to assess the performance of Large language models . they use a set of 15 simple text editing tasks to examine their capabilities .
Outcome:	The proposed benchmark aims to assess the performance of Large language models in basic language tasks.

Copied to clipboard

Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli

GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages (2022.lrec-1)

Challenge:	Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities.
Approach:	They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance.
Outcome:	The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.

Copied to clipboard

Fitsum Gaim, Wonsuk Yang, Jong C. Park

Information Retrieval

Challenge:	Low-resourced languages with similar typologies are often confused with each other in real-world applications such as machine translation, affecting the user’s experience.
Approach:	They propose to build a dataset for five typologically and phylogenetically related low-resourced East African languages using the Ge’ez script as a writing system.
Outcome:	The proposed dataset is built automatically from selected data sources, but also performed a manual evaluation to assess its quality.

Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)

Copied to clipboard

Ioanna Ntinou, Alexandros Xenos, Yassine Ouali, Adrian Bulat, Georgios Tzimiropoulos

Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)

Challenge:	Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour.
Approach:	They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions.
Outcome:	The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries.

Copied to clipboard

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang

Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models (2023.findings-emnlp)

Challenge:	Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks.
Approach:	They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially.
Outcome:	The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks.

Copied to clipboard

Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, Naihao Deng

Bridging Perception, Memory, and Inference through Semantic Relations (2021.emnlp-main)

Challenge:	Theory of Mind (ToM) is the ability to reason about one's own and others' mental states.
Approach:	They propose a higher-order theory of mind benchmark and introduce a new deception mechanism to evaluate ToM reasoning.
Outcome:	The proposed benchmarks show that the LLMs are not performing well on higher-order tasks.

Copied to clipboard

Johanna Björklund, Adam Dahlgren Lindström, Frank Drewes

Language Models are Few-Shot Butlers (2021.emnlp-main)

Challenge:	Recent studies suggest that it is impossible to learn meaning from surface form alone.
Approach:	They propose to develop triadic systems that combine neural and symbolic methods to provide a seamless information flow between them.
Outcome:	The proposed systems combine the strengths of neural and symbolic methods to achieve a seamless information flow between them.

Copied to clipboard

Vincent Micheli, Francois Fleuret

Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models (2023.emnlp-main)

Challenge:	Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets.
Approach:	They propose a two-stage procedure to learn from a small set of demonstrations and a simple reinforcement learning algorithm to improve by interacting with an environment.
Outcome:	The proposed method improves with only 1.2% of the demonstrations and a simple reinforcement learning algorithm over existing methods in the ALFWorld environment.

Copied to clipboard

Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, Yoonsik Kim, Sangdoo Yun, Taeho Kil, Bado Lee, Seunghyun Park

Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison (2023.findings-emnlp)

Challenge:	Recent advances in Large Language Models (LLMs) have stimulated a surge of research aimed at extending their applications to the visual domain.
Approach:	They propose a novel neural architecture to enhance language-image understanding capability of LLMs by capturing intricate details that are often overlooked in existing methods.
Outcome:	The proposed model can achieve better comprehension of language information in visual contexts within images.

Copied to clipboard

Shuo Sun, Yuchen Zhang, Jiahuan Yan, Yuze Gao, Donovan Ong, Bin Chen, Jian Su

Programming Languages in NLP

Challenge:	a number of open-source large language models claim to be performing better than commercial ones . however, these models fall short of the performance achieved by closed-source models like GPT-3.5 .
Approach:	They evaluate six popular large language models against each other to evaluate their performance . authors say open-source models are not as effective as those built by commercial models .
Outcome:	a new set of models claim to match or surpass the language understanding abilities of commercial models . the results show that the models performed far below the performance of closed-source models compared to open-source ones .

Representation Learning with Conditional Information Flow Maximization (2024.acl-long)

Copied to clipboard

Dou Hu, Lingwei Wei, Wei Zhou, Songlin Hu

Human-Inspired Obfuscation for Model Unlearning: Local and Global Strategies with Hyperbolic Representations (2025.findings-emnlp)

Challenge:	Existing knowledge-theoretic representation learning frameworks are based on the information bottleneck principle, which preserves redundant features irrelevant to the given task.
Approach:	They propose a conditional information flow maximization framework to learn sufficient representations for the input data and target task by maximizing both input-representation and representation-label mutual information.
Outcome:	The proposed framework can extract noise-invariant sufficient representations for the input data and target task.

Copied to clipboard

Zekun Wang, Jingjie Zeng, Yingxu Li, Liang Yang, Hongfei Lin

PAD-Net: An Efficient Framework for Dynamic Networks (2023.acl-long)

Challenge:	Existing methods for unlearning large language models struggle to balance effective forgetting with maintaining model utility.
Approach:	They propose a human-inspired unlearning framework that simulates forgetting on fuzzy data and represents them in hyperbolic and Euclidean spaces.
Outcome:	The proposed framework is able to forget sensitive content while maintaining the model’s language understanding, fluency, and benchmark performance.

Copied to clipboard

Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, Dacheng Tao

Green & Sustainable NLP

Challenge:	Dynamic networks can significantly improve the model’s representation power with acceptable computational cost.
Approach:	They propose a partially dynamic network to transform redundant dynamic parameters into static ones and iterative mode partition to partition dynamic and static parameters efficiently.
Outcome:	The proposed network surpasses fully dynamic networks by +0.7% top-1 acc with only 30% dynamic parameters for DY-Conv and +1.9% average score in language understanding with only 50% dynamic parameters.

Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs (2023.findings-emnlp)

Copied to clipboard

Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, Kam-Fai Wong

Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales (2025.findings-acl)

Challenge:	Existing LLMs generate responses based on the dialogue context, overlooking the underlying linguistic cues about the user status exhibited in the context.
Approach:	They propose a linguistic cue-based chain-of-thoughts method which enhances the LLMs inference with an intermediate reasoning step to find cues exhibited in the dialogue.
Outcome:	The proposed method outperforms standard prompting methods on in-depth dialogue questions and linguistic cues exhibited in the context.

Copied to clipboard

Ayuto Tsutsumi, Yuu Jinnai

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models (2026.acl-long)

Challenge:	Large Language Models (LLMs) have strong language understanding and generation abilities, but cultural knowledge is often limited to English-speaking communities.
Approach:	They propose a benchmark dataset to evaluate the cultural awareness of large language models . they use Japanese folktales to evaluate their knowledge of Yokai .
Outcome:	The proposed model trains with Japanese language resources and achieves higher accuracy than English-centric models.

Copied to clipboard

Lin Mu, Haiyang Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024.emnlp-main)

Challenge:	Existing LoRA methods assume that experts operate independently, leading to unstable routing, expert dominance.
Approach:	They propose a communication-aware MoELoRA framework that relaxes this assumption by introducing expert-level communication prior to routing.
Outcome:	The proposed framework outperforms vanilla LoRA and MoELoRA on diverse language understanding tasks while maintaining expert dominance.

Copied to clipboard

Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang

Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers (2023.findings-emnlp)

Challenge:	Referring Expression Comprehension (REC) is a cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding.
Approach:	They propose to use a new reference expression comprehension (REC) dataset to evaluate the capabilities of language understanding, image comprehension, and language-to-image grounding.
Outcome:	The proposed model is able to reject scenarios where the target object is not visible in the image, a key aspect often overlooked in existing models and approaches.

Copied to clipboard

Weng Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Jiahua Liu, Tao Li, Yuxiao Dong, Jie Tang

EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models (2025.findings-emnlp)

Challenge:	Prompt tuning is a technique that updates few parameters in pre-trained models for language understanding and generation tasks.
Approach:	They propose to leverage prompt tuning for neural text retrieval to improve generalization and cross-domain generalization.
Outcome:	The proposed approach can mitigate the two issues faced by fine-tuning retrieval methods and improve the out-of-domain zero-shot generalization of the retrieval models.

Copied to clipboard

Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Kevin Zhu, Austen Liao, Sean O’Brien

HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)

Challenge:	Existing benchmarks often overlook intra-language variations, leaving speakers of non-standard dialects underserved.
Approach:	EnDive evaluates seven state-of-the-art large language models across tasks . human evaluations confirm high translation quality, with average scores of at least 6.02/7 .
Outcome:	EnDive evaluates state-of-the-art large language models across language understanding, reasoning, mathematics, logic tasks.

Copied to clipboard

Amir Cohen, Hilla Merhav-Fine, Yoav Goldberg, Reut Tsarfaty

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)

Challenge:	Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding.
Approach:	They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem.
Outcome:	The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news.

Copied to clipboard

Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella

Challenge:	Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size.
Approach:	They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs .
Outcome:	The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks.

Trick or Neat: Adversarial Ambiguity and Language Model Evaluation (2025.findings-acl)

Copied to clipboard

Antonia Karamolegkou, Oliver Eberle, Phillip Rust, Carina Kauf, Anders Søgaard

Domain-Specific Data Generation Framework for RAG Adaptation (2026.findings-acl)

Challenge:	Direct prompting fails to detect ambiguity while linear probes can decode ambiguities with high accuracy, sometimes exceeding 90%.
Approach:	They introduce an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarials.
Outcome:	The proposed dataset includes syntactic, lexical, and phonological ambiguities along with adversarial variations.

Copied to clipboard

Chris Xing Tian, Weihao Xie, Zhen Chen, Hui Liu, Zhengyuan Yi, Haoliang Li, Shiqi Wang, Siwei Ma

Information Retrieval

Challenge:	Retrieval-Augmented Generation (RAG) combines the language understanding and reasoning capabilities of large language models (LLMs) with external retrieval to produce domain-grounded responses.
Approach:	They propose a scalable and modular data-centric framework for generating domain-grounded question–answer–context triples tailored to diverse RAG adaptation strategies.
Outcome:	The proposed framework generates domain-grounded question–answer–context triples for multiple RAG adaptation strategies.

Impartial Multi-task Representation Learning via Variance-invariant Probabilistic Decoding (2025.acl-long)

Copied to clipboard

Dou Hu, Lingwei Wei, Wei Zhou, Songlin Hu

On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts (2025.emnlp-main)

Challenge:	Existing methods focus on balancing loss or gradients but fail to address this issue due to the representation discrepancy in latent space.
Approach:	They propose a framework that harmonizes representation spaces across tasks to ensure impartial learning by harmonizing representation spaces.
Outcome:	The proposed framework outperforms 12 representative methods under the same multi-task settings, especially in heterogeneous task combinations and data-constrained scenarios.

Copied to clipboard

Linlu Qiu, Cedegao E. Zhang, Joshua B. Tenenbaum, Yoon Kim, Roger P. Levy

Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science (2024.lrec-main)

Challenge:	Language models (LMs) are increasingly used as conversational agents because of their pragmatic reasoning abilities.
Approach:	They propose an evaluation framework derived from Wavelength, a popular communication game where a speaker and a listener communicate about a broad range of concepts in a granular manner.
Outcome:	The proposed evaluation framework outperforms direct and Chain-of-Thought (CoT) prompting on language comprehension and language production tasks.

Copied to clipboard

Yida Mu, Ben P. Wu, William Thorne, Ambrose Robinson, Nikolaos Aletras, Carolina Scarton, Kalina Bontcheva, Xingyi Song

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks (2026.acl-long)

Challenge:	Existing instruction-tuned Large Language Models (LLMs) have impressive language understanding and the capacity to generate responses that follow specific prompts.
Approach:	They evaluate the zero-shot performance of two publicly accessible LLMs, ChatGPT and OpenAssistant, in the context of six Computational Social Science classification tasks.
Outcome:	The proposed LLMs perform better than state-of-the-art models on social science tasks.

Copied to clipboard

Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Banerjee Mohor, Swaagat Bikash Saikia, Alvin Chan

CogBench: Benchmarking Cognitive Alignment of Large Language Models in Educational Question Answering (2026.findings-acl)

Challenge:	Existing methods for evaluating creativity are tightly coupled to specific tasks and limiting scalability and generality.
Approach:	They propose a domain-agnostic framework for quantifying LLM creativity across open-ended tasks.
Outcome:	The proposed framework captures key facets of creativity including novelty, diversity, and task fulfilment with over 60% improved efficiency.

Copied to clipboard

Tong Lu, Zhichun Wang, Yuanhao Sun, Yaoyu Zhou, Mingrui Li, Yiming Guan, Zhiyong Bai

Question Answering

Challenge:	Large language models (LLMs) possess strong capabilities in language understanding and generation, as well as remarkable problem-solving abilities.
Approach:	They propose a benchmark to assess the cognitive alignment capabilities of large language models in educational QA.
Outcome:	The proposed evaluation benchmark assesses the cognitive alignment capabilities of large language models in educational QA.

GraphDx: A Cost-Aware Knowledge-Enhanced Multi-Agent Framework for Sequential Diagnosis (2026.findings-acl)

Copied to clipboard

Shaoting Tan, Ning Liu, Yuntao Du, Shuyue Wei, Wu Shuai, Qian Li, Yanyu Xu, Wei Zhang, Lizhen Cui, Haitao Yuan

TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages (2025.acl-long)

Challenge:	Existing Large Language Models struggle to reason systematically under cost constraints . Existing approaches lack the knowledge-reasoning capability to reason under cost .
Approach:	They propose a knowledge-enhanced framework that leverages large language models to construct MDKGs . they propose three collaborative agents that handle language understanding and generation .
Outcome:	GraphDx improves diagnostic success rates from 50–68% to 79–93% while reducing test costs by 20–54%.

Copied to clipboard

Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, Duygu Ataman

Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention (2025.acl-long)

Challenge:	preparing native language MMLU benchmarks is costly and limits representativeness of evaluation datasets.
Approach:	They propose to use a Turkic language MMLU benchmark to assess massive multitask language understanding capabilities.
Outcome:	The proposed benchmarks are based on a Turkic language morphosyntactic and cultural benchmark . the benchmarks evaluate a diverse range of open and proprietary multilingual large language models .

Copied to clipboard

Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni, Xiaojing Zhao, Xiaoyi Bao

The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)

Challenge:	Autoregressive Large Language Models (LLMs) demonstrate exceptional performance in language understanding and generation tasks, but their application in text embedding tasks has been relatively slow due to the constraints of the unidirectional attention mechanism.
Approach:	They propose to enable bidirectional attention in LLMs to improve their ability to represent subsequent context but weaken their utilization of preceding context.
Outcome:	The proposed model improves the LLMs’ ability to represent subsequent context but weakens their utilization of preceding context, while contrastive learning training can help to maintain both abilities.

Copied to clipboard

Felix Stahlberg, Shankar Kumar

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models (2025.findings-emnlp)

Challenge:	Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections.
Approach:	They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models.
Outcome:	The proposed method is significantly more effective than removing neurons randomly or based on their magnitude.

Copied to clipboard

Xiaojun Wu, Junxi Liu, Huan-Yi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo

SI-NLI: A Slovene Natural Language Inference Dataset and Its Evaluation (2024.lrec-main)

Challenge:	Existing financial benchmarks suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation.
Approach:	They propose a bilingual benchmark for financial LLMs that assesses models’ language understanding and generation capabilities.
Outcome:	The proposed bilingual benchmark assesses models’ language understanding and generation capabilities.

Copied to clipboard

Matej Klemen, Aleš Žagar, Jaka Čibej, Marko Robnik-Šikonja

SynTOD: Augmented Response Synthesis for Robust End-to-End Task-Oriented Dialogue System (2024.lrec-main)

Challenge:	Existing datasets for natural language inference (NLI) are limited to English and a few other well-resourced languages.
Approach:	They propose to use a dataset for natural language inference to extend the resources for the task.
Outcome:	The proposed dataset is constructed from scratch using knowledgeable annotators with carefully crafted guidelines aiming to avoid common problems in existing datasets.

Copied to clipboard

Nguyen Quang Chieu, Quang-Minh Tran, Khac-Hoai Nam Bui

Task-agnostic Distillation of Encoder-Decoder Language Models (2024.lrec-main)

Challenge:	Task-oriented dialogue systems focus on training multiple tasks such as language understanding, tracking states, and generating appropriate responses to help users achieve their specific goals.
Approach:	They exploit the ability of pre-trained models to provide synthesis responses for fine-tuning end-to-end TOD systems.
Outcome:	The proposed model outperforms baseline models on multiwoz datasets and is available for further exploitation.

Copied to clipboard

Chen Zhang, Yang Yang, Qiuchi Li, Jingang Wang, Dawei Song

MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment (2025.findings-acl)

Challenge:	Existing distillation methods that focus on encoder-only LMs fail to handle the distillation of encoder decoder LM.
Approach:	They propose a method that finetunes pretrained language models (LMs) they propose 'MiniEnD' that allows for task-agnostic distillation of LMs.
Outcome:	The proposed distillation method is generally effective and competitive compared to other alternatives.

Copied to clipboard

Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze

Frame First, Then Extract: A Frame-Semantic Reasoning Pipeline for Zero-Shot Relation Triplet Extraction (2025.emnlp-main)

Challenge:	Existing benchmarks for multilinguality for English-centric large language models focus on classic tasks or cover a minimal number of languages.
Approach:	They propose a method to assess multilingual capabilities of pre-trained LLMs using parallel sentences.
Outcome:	The proposed method evaluates the multilingual capabilities of pre-trained English-centric models using parallel sentences.

Copied to clipboard

Zehan Li, Fu Zhang, Wenqing Zhang, Jiawei Li, Zhou Li, Jingwei Cheng, Tianyue Peng

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models (2024.lrec-main)

Challenge:	Existing methods to extract triplets for unseen relations rely on costly fine-tuning and lack structured semantic guidance.
Approach:	They propose a framework that adopts a "frame first, then extract" paradigm to extract triplets from unstructured text.
Outcome:	The proposed framework achieves competitive zero-shot performance on multiple benchmarks and can be used to enhance existing extraction methods.

Copied to clipboard

Junbing Yan, Chengyu Wang, Taolin Zhang, Xiaofeng He, Jun Huang, Wei Zhang, Longtao Huang, Hui Xue

Bayesian Social Deduction with Graph-Informed Language Models (2026.acl-long)

Challenge:	Existing methods for incorporating external knowledge into language models do not prioritize learning embeddings for entity-related tokens.
Approach:	They propose a framework for incorporating external knowledge into pre-training models that utilize entity-related tokens.
Outcome:	The proposed framework reduces pre-training time by 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks.

Copied to clipboard

Shahab Rahimirad, Guven Gergerli, Lucia Romero, Angela Qian, Matthew Lyle Olson, Simon Stepputtis, Joseph Campbell

Which Sense Dominates Multisensory Semantic Understanding? A Brain Decoding Study (2024.lrec-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable general-purpose reasoning capabilities across a wide range of tasks.
Approach:	They propose a hybrid reasoning framework that externalizes belief inference to a structured probabilistic model while using an LLM for language understanding and interaction.
Outcome:	The proposed framework achieves competitive performance with larger models in Agent-Agent play and is the first language agent to defeat human players in a controlled study.

Copied to clipboard

Dandan Huang, Lu Cao, Zhenting Li, Yue Zhang

WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations (2026.findings-acl)

Challenge:	Decoding semantic meanings from brain activity is open to multisensory stimulation, as word meanings can be delivered by both auditory and visual inputs.
Approach:	They aim to develop a computational model to probing what information from the act of language understanding is represented in human brain.
Outcome:	The proposed model dissociates multisensory integration of word understanding into written text, spoken text and image perception respectively, exploring the decoding efficiency and reliability of unisensory information in the brain representation.

Copied to clipboard

Jaeyeon Kim, Heeseung Yun, Tony Woo, Chao-Han Huck Yang, Gunhee Kim

Speech & Audio in NLP

Challenge:	Large audio-language models extend language understanding into the auditory domain, yet their ability to perform low-level listening, such as pitch and duration detection, remains underexplored.
Approach:	They propose a global benchmark to evaluate low-level auditory perception and cognition using marine mammal vocalizations to better assess models’ low- level listening.
Outcome:	The proposed models show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.

Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning (2025.acl-long)

Copied to clipboard

Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty

Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models? (2026.findings-acl)

Challenge:	Existing methods to train student models on the generated outputs of teacher models are not efficient for ICL.
Approach:	They propose to align the output of smaller (student) models with that of larger (teacher) models by incorporating a ranking loss and aligning the token-level output distribution.
Outcome:	The proposed model outperforms baseline models on a variety of tasks involving language understanding, reasoning, and coding.

Copied to clipboard

Deokhyung Kang, Seonjeong Hwang, Daehui Kim, Hyounghun Kim, Gary Lee

LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation (2026.acl-long)

Challenge:	Recent reasoning language models (RLMs) achieve strong performance on complex reasoning tasks, yet they still exhibit a multilingual reasoning gap.
Approach:	They propose a strategy that incorporates an English translation into the initial reasoning trace when an understanding failure is detected.
Outcome:	The proposed strategy incorporates an English translation into the initial reasoning trace when an understanding failure is detected.

Copied to clipboard

Siqing Song, Chuang Wang, Yong Lang, Yi Yang, Xu-Yao Zhang

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)

Challenge:	Existing methods for implementing large language models are limited by high computational and memory requirements.
Approach:	They propose a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy.
Outcome:	The proposed framework surpasses state-of-the-art methods on W2A4 quantization settings across languages.

Copied to clipboard

Pavel Chizhov, Egor Bogomolov, Ivan P. Yamshchikov

CoDial: Interpretable Task-Oriented Dialogue Systems Through Dialogue Flow Alignment (2026.acl-long)

Challenge:	Currently, subword tokenization is the most common approach for vocabulary building in large models.
Approach:	They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens .
Outcome:	The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE.

Copied to clipboard

Radin Shayanfar, Chu Fei Luo, Rohan V Bhambhoria, Samuel Dahan, Xiaodan Zhu